Data-Driven Feedback Control System for Real-Time Application Support in Virtualized Networks

ABSTRACT

Concepts and technologies disclosed herein are directed to data-driven feedback control systems for an acceptable level of real-time application transaction completion rate in virtualized networks, while maximizing virtualized server utilization. According to one aspect disclosed herein, a network virtualization platform (“NVP”) includes a plurality of hardware resources, a virtual machine (“VM”), and a virtual machine monitor (“VMM”). The VMM can track an execution state of each of a plurality of applications associated with the VM. The VMM can measure a real-time application transaction completion rate of the VM. The VMM can determine whether a trigger condition exists for priority scheduling of real-time applications based upon the real-time application transaction completion rate and a pre-set threshold value. The VMM can, in response to determining that the trigger condition exists, apply a priority control schedule to instruct the VM to perform priority processing of a real-time application over a non-real-time application.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 14/806,906, entitled “Data-Driven Feedback ControlSystem for Real-Time Application Support in Virtualized Networks,” filedJul. 23, 2015, now allowed, which is incorporated herein by reference inits entirety.

BACKGROUND

Cloud computing can allow dynamically scalable virtualized resources tobe provided as a service. Cloud computing can assure an appropriatelevel of resources are available to power software applications when andwhere the resources are needed in response to demand. As a result, cloudcomputing allows entities to respond quickly, efficiently and in anautomated fashion to the rapidly changing business environment.

Real-time and virtualization by design are at odds with each other.Virtualization spreads computing resources across multiple virtualmachines as a way to increase server utilization and flexibility.Real-time applications dedicate specific resources to reduce computingoverhead and to ensure low latency. Currently, network virtualizationplatforms (“NVPs”) utilize fair scheduling and load balancing tomaximize server utilization. Fair scheduling and load balancingfunctions do not support real-time application performance requirementswell during capacity reduction and/or network overload events. Thus,both technical and business challenges exist for how to optimizetradeoffs between maximum virtualized server utilization and anacceptable level of real-time application transaction completion rate.

SUMMARY

Concepts and technologies disclosed herein are directed to a data-drivenfeedback control system that achieves an acceptable level of real-timeapplication transaction completion rate in virtualized networks, whilemaximizing virtualized server utilization. Moreover, the concepts andtechnologies disclosed herein can be utilized to at least meet one ormore service level agreements (“SLAs”) when network capacity is strainedor in an overload condition.

According to one aspect disclosed herein, a network virtualizationplatform (“NVP”) includes a plurality of hardware resources, one or morevirtual machines (“VMs”), and one or more virtual machine monitors(“VMMs”), which can include one or more real-time VMMs. In someembodiments, the NVP includes a plurality of server clusters, whereineach server cluster includes a plurality of hardware resources, aplurality of VMs, and a real-time VMM. The VMM can include a pluralityof data-driven feedback control modules that, when executed by a computeresource of the plurality of hardware resources, causes the VMM toperform operations. In particular, the VM can apply fair schedulingcontrol for all applications as part of a baseline scheduling controlscheme. The VMM can track an execution state of each of a plurality ofapplications associated with the VM, including real-time applicationsand non-real-time applications. The VMM can measure a real-timeapplication transaction completion rate of the VM. The VMM can determinewhether a trigger condition exists for priority scheduling of real-timeapplications based upon the real-time application transaction completionrate and a pre-set real-time transaction completion rate thresholdvalue. The VMM can, in response to determining that the triggercondition exists, apply a priority control schedule to instruct the VMto perform priority processing of a real-time application of theplurality of applications over a non-real-time application of theplurality of applications. The VMM can reapply the baseline fairscheduling control scheme when the trigger condition is determined to becleared in order to maximize virtualized server utilization.

In some embodiments, the plurality of data-driven feedback controlmodules includes a VM fair scheduling module. In these embodiments, theVMM can apply the baseline scheduling control scheme to instruct the VMto perform baseline processing of the plurality of applications,including real-time and non-real-time applications.

In some embodiments, the plurality of data-driven feedback controlmodules includes a real-time application execution state monitoringmodule. In these embodiments, the VMM can determine whether anapplication transaction of the real-time application is in a processingqueue for execution. In response to determining that the applicationtransaction of the real-time application is in a processing queue forexecution, the VMM can record a state change from a ready/wait executionstate to a run execution state. The VMM also can determine whether theapplication transaction of the real-time application completedsuccessfully. If the application transaction completed successfully, theVMM can increment a successful application transaction completioncounter and can report the successful application transaction completioncounter to a priority scheduling trigger module of the plurality ofdata-driven feedback control modules. If the application transaction didnot complete successfully, the VMM can increment an unsuccessfulapplication transaction completion counter and can report theunsuccessful application transaction completion counter to the priorityscheduling trigger module of the plurality of data-driven feedbackcontrol modules. A total real-time application transaction count in agiven time period can be measured by summing the applicationtransactions that completed successfully and the applicationtransactions that completed unsuccessfully in the given time period.

In some embodiments, the plurality of data-driven feedback controlmodules includes a priority scheduling trigger module and a priorityscheduling control module. In these embodiments, the VMM can receive thesuccessful application transaction completion counter and theunsuccessful application transaction completion counter from thereal-time application execution state monitoring module. The VMM candetermine a critical condition. The critical condition can be expressedas a real-time application transactions completion rate below a pre-setthreshold value. The VMM can, in response to determining that thecritical condition exists, notify the priority scheduling control moduleof the critical condition.

In some embodiments, the VMM can receive notification from the priorityscheduling trigger module that a critical condition exists. In responseto receiving notification from the priority scheduling trigger module,the VMM can apply the priority control schedule to instruct the VMM toperform priority processing of the real-time application of theplurality of applications over the non-real-time application of theplurality of applications.

In some embodiments, the VMM can reset a first execution state of thenon-real-time application on the VM to ready/wait. The VMM also can seta second execution state of the real-time application on the virtualmachine to run. In some embodiments, the VMM can reset the firstexecution state prior to a time-out.

It should be appreciated that the above-described subject matter may beimplemented as a computer-controlled apparatus, a computer process, acomputing system, or as an article of manufacture such as acomputer-readable storage medium. These and various other features willbe apparent from a reading of the following Detailed Description and areview of the associated drawings.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intendedthat this Summary be used to limit the scope of the claimed subjectmatter. Furthermore, the claimed subject matter is not limited toimplementations that solve any or all disadvantages noted in any part ofthis disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating aspects of an illustrativeoperating environment for implementing the various concepts andtechnologies disclosed herein.

FIG. 2 is a block diagram illustrating aspects of an illustrativedata-driven feedback control model framework for implementing thevarious concepts and technologies disclosed herein.

FIG. 3 is a flow diagram illustrating aspects of a method for utilizinga data-driven feedback control model to perform operations, according toan illustrative embodiment.

FIGS. 4A-4B are flow diagrams illustrating aspects of a method forexecuting an application execution state monitoring module to performstate monitoring functions, according to an illustrative embodiment.

FIG. 5 is a flow diagram illustrating aspects of a method for executinga priority scheduling trigger module to perform priority schedulingtrigger functions, according to an illustrative embodiment.

FIG. 6 is a flow diagram illustrating aspects of a method for executinga priority scheduling control module to perform priority schedulingfunctions, according to an illustrative embodiment.

FIG. 7 is a block diagram illustrating an example computer systemcapable of implementing aspects of the embodiments presented herein.

FIG. 8 is a diagram illustrating a software defined network (“SDN”),according to an illustrative embodiment.

DETAILED DESCRIPTION

Real-time and virtualization by design are at odds with each other.Virtualization spreads computing resources across multiple virtualmachines as a way to increase server utilization and flexibility.Real-time applications dedicate specific resources to reduce computingoverhead and to ensure low latency. Currently, network virtualizationplatforms utilize fair scheduling and load balancing to maximize serverutilization. Fair scheduling and load balancing functions do not supportreal-time application performance requirements well during capacityreduction and/or network overload events. Thus, both technical andbusiness challenges exist for how to optimize tradeoffs between maximumvirtualized server utilization and an acceptable level of real-timeapplication transaction completion rate.

The concepts and technologies disclosed herein are directed to aclosed-loop feedback control system that maintains maximumvirtualization benefits (server utilization) subject to an acceptablelevel of real-time applications transaction completion rate in a giventime period. The feedback control closed-loop model integratesmonitoring strategy, data analytics, and application scheduling controlvia a closed-loop sequential flow of real-time application performancemonitoring, key performance indicator (“KPI”) measurements, KPIviolation feedback control triggering, real-time priority controlinitiation, and real-time feedback data monitoring and measurement. Asused herein, “real-time’ is the actual time during which monitoring isperformed.

User-defined, on-demand cloud services and user digital experienceexpectations are driving planning and deployment of network functionalvirtualization and service-centric software-defined networking amongglobal telecommunications service providers. Network VirtualizationPlatforms (“NVPs”) are deployed in information technology (“IT”) datacenters and network central offices to accelerate deployment ofon-demand user service and virtualized network functions. An NVP is ashared virtualized infrastructure that supports multiple services andnetwork applications (including real-time and non-real-timeapplications). However, the current design of virtualization is tomaximize server utilization, which could significantly affect support ofreal-time applications when network capacity is in strained conditionsor the network is in overload conditions. This can affect the adoptionof NVP and slow down wide-spread commercial offerings of user-definedcloud services and deployment of software-centric cloud networkinginfrastructure.

A benefit of the concepts and technologies disclosed herein is theability to maximize NVP serviceable potential of being deployed inuser-defined cloud service environments supporting both real-time andnon-real-time applications without changing current fair-basedscheduling design of NVP. The concepts and technologies disclosed hereinprovide a data-driven heuristic model utilized to maximizevirtualization benefits (server utilization) subject to a real-timeapplication transaction completion rate to an acceptable level via aclosed loop feedback-control model. The concepts and technologiesdisclosed herein also provide a feedback control model utilized tointegrate a closed-loop sequential flow of real-time applicationperformance monitoring, KPI measurements, KPI violation feedback controltriggering, real-time priority control initiation, and real-timefeedback data monitoring and measurement.

While the subject matter described herein may be presented, at times, inthe general context of program modules that execute in conjunction withthe execution of an operating system and application programs on acomputer system, those skilled in the art will recognize that otherimplementations may be performed in combination with other types ofprogram modules. Generally, program modules include routines, programs,components, data structures, computer-executable instructions, and/orother types of structures that perform particular tasks or implementparticular abstract data types. Moreover, those skilled in the art willappreciate that the subject matter described herein may be practicedwith other computer systems, including hand-held devices, mobiledevices, wireless devices, multiprocessor systems, distributed computingsystems, microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers, routers, switches, other computingdevices described herein, and the like.

Referring now to FIG. 1, aspects of an operating environment 100 forimplementing various embodiments of the concepts and technologiesdisclosed herein will be described. The illustrated operatingenvironment 100 includes a network virtualization platform (“NVP”) 102.The NVP 102 is a shared infrastructure that can support multipleservices and network applications, including one or more non-real-timeapplications 104 and one or more real-time applications 106. Thereal-time applications 106 can include, but are not limited to, one ormore applications that provide or utilize, at least in part, domain nameservice (“DNS”), remote authentication dial-in user service (“RADIUS”),dynamic host configuration protocol (“DHCP”), lightweight directoryaccess protocol (“LDAP”), content routing in either or both of thecontrol and data planes, and route reflecting in either or both of thecontrol and data planes. The real-time applications 106 also can includeone or more applications that provide, at least in part, one or morereal-time services such as, but are not limited to, voice over internetprotocol (“VoIP”) service, streaming video service, videoconferencingservice, online gaming service, chatting service, instant messaging(“IM”) service, and the like in the service plane.

The illustrated NVP 102 includes a hardware resource layer 108, avirtualization/control layer 110, and a virtual resource layer 112 thatwork together to perform operations as will be described in detailherein. While connections are shown between some of the componentsillustrated in FIG. 1, it should be understood that some, none, or allof the components illustrated in FIG. 1 can be configured to interactwith one another to carry out various functions described herein. Insome embodiments, the components are arranged so as to communicate viaone or more networks (best shown in FIG. 8). Thus, it should beunderstood that FIG. 1 and the following description are intended toprovide a general understanding of a suitable environment in whichvarious aspects of embodiments can be implemented, and should not beconstrued as being limiting in any way.

The hardware resource layer 108 provides hardware resources, which, inthe illustrated embodiment, include one or more compute resources 114,one or more memory resources 116, and one or more other resources 118.The compute resource(s) 114 can include one or more hardware componentsthat perform computations to process data, and/or to executecomputer-executable instructions of one or more application programs,operating systems, and/or other software. The compute resources 114 caninclude one or more central processing units (“CPUs”) configured withone or more processing cores. The compute resources 114 can include oneor more graphics processing unit (“GPU”) configured to accelerateoperations performed by one or more CPUs, and/or to perform computationsto process data, and/or to execute computer-executable instructions ofone or more application programs, operating systems, and/or othersoftware that may or may not include instructions particular to graphicscomputations. In some embodiments, the compute resources 114 can includeone or more discrete GPUs. In some other embodiments, the computeresources 114 can include CPU and GPU components that are configured inaccordance with a co-processing CPU/GPU computing model, wherein thesequential part of an application executes on the CPU and thecomputationally-intensive part is accelerated by the GPU. The computeresources 114 can include one or more system-on-chip (“SoC”) componentsalong with one or more other components, including, for example, one ormore of the memory resources 116, and/or one or more of the otherresources 118. In some embodiments, the compute resources 114 can be orcan include one or more SNAPDRAGON SoCs, available from QUALCOMM of SanDiego, Calif.; one or more TEGRA SoCs, available from NVIDIA of SantaClara, Calif.; one or more HUMMINGBIRD SoCs, available from SAMSUNG ofSeoul, South Korea; one or more Open Multimedia Application Platform(“OMAP”) SoCs, available from TEXAS INSTRUMENTS of Dallas, Tex.; one ormore customized versions of any of the above SoCs; and/or one or moreproprietary SoCs. The compute resources 114 can be or can include one ormore hardware components architected in accordance with an ARMarchitecture, available for license from ARM HOLDINGS of Cambridge,United Kingdom. Alternatively, the compute resources 114 can be or caninclude one or more hardware components architected in accordance withan x86 architecture, such an architecture available from INTELCORPORATION of Mountain View, Calif., and others. Those skilled in theart will appreciate the implementation of the compute resources 114 canutilize various computation architectures, and as such, the computeresources 114 should not be construed as being limited to any particularcomputation architecture or combination of computation architectures,including those explicitly disclosed herein.

The memory resource(s) 116 can include one or more hardware componentsthat perform storage operations, including temporary or permanentstorage operations. In some embodiments, the memory resource(s) 116include volatile and/or non-volatile memory implemented in any method ortechnology for storage of information such as computer-readableinstructions, data structures, program modules, or other data disclosedherein. Computer storage media includes, but is not limited to, randomaccess memory (“RAM”), read-only memory (“ROM”), Erasable ProgrammableROM (“EPROM”), Electrically Erasable Programmable ROM (“EEPROM”), flashmemory or other solid state memory technology, CD-ROM, digital versatiledisks (“DVD”), or other optical storage, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store data and which can be accessedby the compute resources 114.

The other resource(s) 118 can include any other hardware resources thatcan be utilized by the compute resources(s) 114 and/or the memoryresource(s) 116 to perform operations described herein. The otherresource(s) 118 can include one or more input and/or output processors(e.g., network interface controller or wireless radio), one or moremodems, one or more codec chipset, one or more pipeline processors, oneor more fast Fourier transform (“FFT”) processors, one or more digitalsignal processors (“DSPs”), one or more speech synthesizers, and/or thelike.

The hardware resources operating within the hardware resources layer 108can be virtualized by one or more virtual machine monitors (“VMMs”)120-120K (also known as “hypervisors”; hereinafter “VMMs 120”) operatingwithin the virtualization/control layer 110 to manage one or morevirtual resources that reside in the virtual resource layer 112. TheVMMs 120 can be or can include software, firmware, and/or hardware thatalone or in combination with other software, firmware, and/or hardware,manages one or more virtual resources operating within the virtualresource layer 112. In the illustrated embodiment, the VMMs 120 canexecute one or more data-driven feedback control modules 122 to performfeedback control operations disclosed herein. More particularly, thedata-driven feedback control modules 122 can be executed by one or moreof the VMMs 120 to perform operations to integrate monitoring strategy,data analytics and application scheduling control via a close-loopsequential flow of real-time application performance monitoring, KPImeasurements, KPI violation feedback control triggering, real-timepriority control initiation, and real-time feedback data monitoring andmanagement. Additional details in this regard will be described hereinbelow with reference to FIG. 2, which provides an overview of severalillustrative examples of the data-driven feedback control modules 122,and with further reference to FIGS. 3-6, which describe specificoperations each of the modules might perform.

The virtual resources operating within the virtual resource layer 112can include abstractions of at least a portion of the compute resources114, the memory resources 116, the other resources 118, or anycombination thereof. These abstractions are referred to herein asvirtual machines (“VMs”). In the illustrated embodiment, the virtualresource layer 112 includes VMs 124-124N (hereinafter “VMs 124”). TheVMs 124 can execute the non-real-time applications 104 and the real-timeapplications 106. Each of the VMs 124 can execute multiple applications,including any combination of the non-real-time applications 104 and thereal-time applications 106. For example, one or more of the VMs 124 canexecute one or more of the real-time applications 106 to utilize orprovide, at least in part, one or more services such as, but not limitedto, DNS, RADIUS, DHCP, LDAP, content routing in either or both of thecontrol and data planes, and route reflecting in either or both thecontrol and data planes. The real-time applications 106 also can includeone or more applications that provide, at least in part, one or morereal-time services such as, but are not limited to, VoIP, streamingvideo service, videoconferencing service, online gaming service,chatting service, IM service, and the like in the service plane.

In some embodiments, a server can include a plurality of serverclusters. A server cluster can include one or more of the VMs 124, oneor more of the VMMs 120, and a plurality of host hardware resources,such as one or more of the computing resources 114, one or more of thememory resources 116, and one or more of the other resources 118.

Turning now to FIG. 2, an illustrative example of a data-driven feedbackcontrol model framework 200, including the data-driven feedback controlmodules 122 introduced in FIG. 1, will be described in context ofexecution via one of the VMMs 120. In the illustrated embodiment, thedata-driven feedback control model framework 200 includes a VM fairscheduling control module 202, a real-time application execution statemonitoring module 204, a priority scheduling trigger module 206, and apriority scheduling control module 208. In summary, the data-drivenfeedback control model framework 200 performs functions, including:applying fair scheduling control as a baseline scheduling controlscheme; monitoring real-time application throughput data associated withthe operation of the real-time applications 106; measuring KPI for usein computing real-time application transaction completion rates;triggering priority scheduling control; initiating priority schedulingcontrol for real-time application transactions; and returning to fairscheduling control. Each of the data-driven feedback control modules 122of the illustrated data-driven feedback control model framework 200 willnow be described in detail.

The VM fair scheduling control module 202 provides virtualizationcontrol functions to control virtualization of resources in the hardwareresource layer 108 in accordance with a normal mode of schedulingcontrol. In some embodiments, the VM fair scheduling control module 202is or includes a baseline hypervisor scheduler that provides defaultscheduling to maximize resource utilization. In other words, the VM fairscheduling control module 202 provides fairness among the availablehardware resources to maximize resource utilization. As explained above,real-time and virtualization by design are at odds with each other.Virtualization spreads computing resources across multiple VMs as a wayto increase server utilization and flexibility. Real-time applicationsdedicate specific resources to reduce computing overhead and to ensurelow latency. The VM fair scheduling control module 202 can implementfair scheduling and load balancing to maximize resource utilization.Fair scheduling and load balancing functions do not support real-timeapplication performance requirements well during capacity reductionand/or network overload events. For this reason, the other modules ofthe illustrated data-driven feedback control model framework 200facilitate priority mode scheduling for one or more of the real-timeapplications 106 as an exception.

The VM fair scheduling control module 202 provides a baseline controlschedule to control the virtualization of resources for executing, atleast in part, the non-real-time-applications 104 and the real-timeapplications 106 under normal operating conditions. For example, athreshold value can be established for a real-time applicationtransaction completion rate such that when a completion rate ofreal-time application transactions during execution in a given timeperiod is detected below the threshold value, the priority schedulingcontrol module 208 can be triggered to overtake control of scheduling ofresource virtualization. As used herein, a “real-time applicationtransaction completion rate” is defined as a percentage of real-timeapplication transactions successfully completed divided by a totalnumber of application transactions being processed in a given timeperiod. If and when the real-time application transaction rate returnsto above the threshold value in a pre-determined measure time interval,the VM fair scheduling control module 202 can retake control ofscheduling of resource virtualization. Each of the real-timeapplications 106 can be assigned one or more threshold values forreal-time application transaction rate. It is contemplated that thethreshold values might change based upon time of day and/or otherfactors.

The real-time application execution state monitoring module 204 tracksexecution states of one or more of the VMs 124 in real-time and markstime stamps for when the execution state changes. The applicationexecution states can include a run execution state, a ready executionstate, and a wait execution state. The run execution state indicatesthat an application (e.g., one of the non-real-time applications 104 orthe real-time applications 106), or at least a portion thereof, iscurrently in execution by one or more of the VMs 124. The readyexecution state indicates that an application (e.g., one of thenon-real-time applications 104 or the real-time applications 106), or atleast a portion thereof, is in a processing queue for execution by oneor more of the VMs 124 but is not currently in execution. The waitexecution state indicates that an application (e.g., one of thenon-real-time applications 104 or the real-time applications 106), or atleast a portion thereof, is not in the processing queue of one or moreof the VMs 124 and is not currently in execution. Additional detailsregarding functions performed by the real-time application executionstate monitoring module 204 are provided below with reference to FIGS.4A-4B.

The priority scheduling trigger module 206 triggers priority schedulingwhen a critical condition for real-time application transactions isdetected. A critical condition can be expressed as a real-timeapplication transactions completion rate below a pre-set thresholdvalue, where the real-time application transaction completion rate isequal to a number of real-time application transactions beingsuccessfully processed divided by a number of real-time applicationtransactions entering into a host resource cluster in a given timeperiod.

The priority scheduling control module 208 can be triggered by thepriority scheduling trigger module 206 when a critical condition forreal-time application transactions is detected. The priority schedulingcontrol module 208 can reset the run execution state of one or more ofthe VMs 124 executing one or more of the non-real-time applications 104to the ready execution state before time-out (e.g., 50 milliseconds) viaan interrupt request or signal. The priority scheduling control module208 can set one or more of the real-time applications 106 queued on oneor more of the VMs 124 to the run execution state.

Turning now to FIG. 3, aspects of a method 300 for implementing thedata-driven feedback control model framework 200 by one of the VMMs 120,according to an illustrative embodiment. It should be understood thatthe operations of the methods disclosed herein are not necessarilypresented in any particular order and that performance of some or all ofthe operations in an alternative order(s) is possible and iscontemplated. The operations have been presented in the demonstratedorder for ease of description and illustration. Operations may be added,omitted, and/or performed simultaneously, without departing from thescope of the concepts and technologies disclosed herein.

It also should be understood that the methods disclosed herein can beended at any time and need not be performed in its entirety. Some or alloperations of the methods, and/or substantially equivalent operations,can be performed by execution of computer-readable instructions includedon a computer storage media, as defined herein. The term“computer-readable instructions,” and variants thereof, as used herein,is used expansively to include routines, applications, applicationmodules, program modules, programs, components, data structures,algorithms, and the like. Computer-readable instructions can beimplemented on various system configurations including single-processoror multiprocessor systems, minicomputers, mainframe computers, personalcomputers, hand-held computing devices, microprocessor-based,programmable consumer electronics, combinations thereof, and the like.

Thus, it should be appreciated that the logical operations describedherein are implemented (1) as a sequence of computer implemented acts orprogram modules running on a computing system and/or (2) asinterconnected machine logic circuits or circuit modules within thecomputing system. The implementation is a matter of choice dependent onthe performance and other requirements of the computing system.Accordingly, the logical operations described herein are referred tovariously as states, operations, structural devices, acts, or modules.These states, operations, structural devices, acts, and modules may beimplemented in software, in firmware, in special purpose digital logic,and any combination thereof. As used herein, the phrase “cause aprocessor to perform operations” and variants thereof is used to referto causing a processor of the NVP 102, such as one or more of thecompute resources 114, and/or a processor one or more other computingsystems and/or devices disclosed herein to perform operations.

For purposes of illustrating and describing some of the concepts of thepresent disclosure, the methods disclosed herein are described as beingperformed, at least in part, by one or more of the VMMs 120 executed byone or more of the hardware resources of the hardware resource layer 108described herein above with reference to the illustrative NVP 102 inFIG. 1, via execution of one or more software modules, which may beembodied, at least in part, by the data-driven feedback control modules122 of the data-driven feedback control model framework 200 describedabove with reference to FIG. 2. It should be understood that additionaland/or alternative devices and/or network nodes can provide thefunctionality described herein via execution of one or more modules,applications, and/or other software. Thus, the illustrated embodimentsare illustrative, and should not be viewed as being limiting in any way.

The method 300 will be described with reference to FIG. 3 and furtherreference to FIGS. 1 and 2. The method 300 begins at operation 302,where the VMM 120 executes the VM fair scheduling control module 202 toperform scheduling control functions. The scheduling control functionscan include controlling virtualization of resources in the hardwareresource layer 108 in accordance with a normal mode of schedulingcontrol. The VM fair scheduling control module 202, as described above,provides a baseline control schedule to control the virtualization ofresources for executing, at least in part, thenon-real-time-applications 104 and the real-time applications 106 undernormal operating conditions. For example, a threshold value can beestablished for a real-time application transaction completion rate suchthat when a transaction completion rate of a real-time application, in agiven time period, is detected below the threshold value, the priorityscheduling control module 208 can be triggered to overtake control ofscheduling of resource virtualization. If and when the real-timeapplication transaction completion rate returns to above the thresholdvalue in a pre-set measured time period, the VM fair scheduling controlmodule 202 can retake control of scheduling of resource virtualization.Each of the real-time applications 106 can be assigned one or morethreshold values for the real-time application transaction completionrate. It is contemplated that the applicable transaction completion ratevalue might change based upon time of day and/or other factors.

From operation 302, the method 300 proceeds to operation 304, where theVMM 120 executes the real-time application execution state monitoringmodule 204 to perform state monitoring functions. The state monitoringfunctions can include tracking application execution states on the VMs124 and marking time stamps for execution state changes. The applicationexecution states can include a run execution state, a ready executionstate, and a wait execution state, each of which is described above.

From operation 304, the method 300 proceeds to operation 306, where theVMM 120 executes the priority scheduling trigger module 206 to performtrigger detection and resolution functions. The trigger detection andresolution functions can include detecting a critical condition. Acritical condition can be expressed as the real-time applicationtransactions completion rate in a given time period drops below apre-set threshold value, where the real-time application transactioncompletion rate is equal to a number of real-time applicationtransactions being successfully processed divided by a number ofreal-time application transactions entering into a host cluster in agiven time period. In particular, from operation 306, the method 300proceeds to operation 308, where the VMM 120 executes the priorityscheduling trigger module 206 to determine whether one or moreperformance trigger conditions have been met. If not, the method 300returns to operation 302. If so, the method 300 proceeds to operation310, where the VMM 120 executes the priority scheduling trigger module206 to determine whether a subject real-time application is presently inqueue for processing. If not, the method 300 returns to operation 304.If so, the priority scheduling trigger module 206 triggers the priorityscheduling control module 208 and the method 300 proceeds to operation312.

At operation 312, the VMM 120 executes the priority scheduling controlmodule 208 to perform priority scheduling functions for the subjectreal-time application 106. The priority scheduling functions can includeresetting the run execution state of one or more of the VMs 124executing one or more of the non-real-time applications 104 to the readyexecution state before time-out (e.g., 50 milliseconds) via an interruptrequest or signal. The priority scheduling control module 208 can setone or more of the subject real-time applications 106 queued on one ormore of the VMs 124 to the run execution state. From operation 312, themethod 300 can return to operation 304.

Turning now to FIGS. 4A-4B, a method 400 for executing the real-timeapplication execution state monitoring module 204 will be described indetail, according to an illustrative embodiment. The method 400 will bedescribed from the perspective of one of the VMMs 120 executing thereal-time application execution state monitoring module 204 to performvarious operations. Moreover, the method 400 will be described withreference to FIGS. 4A-4B and further reference to FIGS. 1 and 2.

The method 400 begins and proceeds to operation 402, where the real-timeapplication execution state monitoring module 204 waits for aninstruction to track one or more applications, such as one or more ofthe real-time applications 106. From operation 402, the method 400proceeds to operation 404, where the real-time application executionstate monitoring module 204 determines whether an instruction to trackone or more applications has been received. If no instruction isreceived, the method 400 returns to operation 402. If an instruction totrack one or more applications is received, the method 400 proceeds tooperation 406, where the real-time application execution statemonitoring module 204 tracks the execution state of the application(s)identified in the instruction. The execution state can be run, ready, orwait. The ready and wait execution states are execution statesindicative of an application transaction that is pending execution.

From operation 406, the method 400 proceeds to operation 408, where thereal-time application execution state monitoring module 204 determineswhether a real-time application transaction is pending execution. Ifnot, the method 400 returns to operation 406. If so, the method 400proceeds to operation 410, where the real-time application executionstate monitoring module 204 records a state change from a ready or waitexecution state to a run state.

Turning now to FIG. 4B, the method 400 proceeds from operation 410 tooperation 412, where the real-time application execution statemonitoring module 204 determines whether the application transaction hascompleted successfully. If so, the method 400 proceeds to operation 414,where the real-time application execution state monitoring module 204increments a successful application transaction completion counter andreports to the priority scheduling trigger module 206. If not, themethod 400 proceeds to operation 416, where the real-time applicationexecution state monitoring module 204 increments an unsuccessfulapplication transaction completion counter and reports to the priorityscheduling trigger module 206. The real-time application execution statemonitoring module 204 counts a total number of real-time applicationtransactions in a given period by summing the number of real-timeapplication transactions completed successfully and the number ofreal-time application transactions completed unsuccessfully in the giventime period. From operation 416, the method 400 proceeds to operation418, where the method 400 ends.

Turning now to FIG. 5, a method 500 for executing the priorityscheduling trigger module 206 will be described in detail, according toan illustrative embodiment. The method 500 will be described from theperspective of one of the VMMs 120 executing the priority schedulingtrigger module 206 to perform various operations. Moreover, the method500 will be described with reference to FIG. 5 and further reference toFIGS. 1 and 2.

The method 500 begins and proceeds to operation 502, where the priorityscheduling trigger module 206 receives the successful applicationtransaction completion counter and the total real-time applicationtransaction counter entering into a host server cluster in a given timeperiod from the real-time application execution state monitoring module204. From operation 502, the method 500 proceeds to operation 504, wherethe priority scheduling trigger module 206 determines a criticalcondition for real-time application transactions.

From operation 504, the method 500 proceeds to operation 506, where thepriority scheduling trigger module 206 determines whether the criticalcondition determined at operation 504 has been detected. If so, themethod 500 proceeds to operation 508, where the priority schedulingtrigger module 206 notifies the priority scheduling control module 208of the critical condition. From operation 508, the method 500 proceedsto operation 510, where the method 500 ends. If not, the method 500proceeds directly to operation 510.

Turning now to FIG. 6, a method 600 for executing the priorityscheduling control module 208 for real-time application transactionswill be described in detail, according to an illustrative embodiment.The method 600 will be described from the perspective of one of the VMMs120 executing the priority scheduling control module 208 to performvarious operations. Moreover, the method 600 will be described withreference to FIG. 6 and further reference to FIGS. 1 and 2.

The method 600 begins and proceeds to operation 602, where the priorityscheduling control module 208 receives notification from the priorityscheduling trigger module 206 to enter a priority scheduling mode forreal-time application transactions. From operation 602, the method 600proceeds to operation 604, where the priority scheduling control module208 resets the run state of the VM running any of the non-real-timeapplications 104 to a ready state before a time-out expires via aninterrupt request or signal.

From operation 604, the method 600 proceeds to operation 606, where thepriority scheduling control module 208 sets the queued real-timeapplication on the VM to the run state. From operation 606, the method600 proceeds to operation 608. The method 600 ends at operation 608.

FIG. 7 is a block diagram illustrating a computer system 700 configuredto provide the functionality in accordance with various embodiments ofthe concepts and technologies disclosed herein. In some implementations,at least a portion of the hardware resources in the hardware resourcelayer 108 (best illustrated in FIG. 1) are provided, at least in part,by one or more host server computers (collectively, “host servercluster”) is configured like the architecture of the computer system700. It should be understood, however, that modification to thearchitecture may be made to facilitate certain interactions amongelements described herein. In some implementations, the computeresources 114, the memory resources 116, and/or the other resources 118are configured like the architecture of the computer system 700 orportions thereof.

The computer system 700 includes a processing unit 702, a memory 704,one or more user interface devices 706, one or more input/output (“I/O”)devices 708, and one or more network devices 710, each of which isoperatively connected to a system bus 712. The bus 712 enablesbi-directional communication between the processing unit 702, the memory704, the user interface devices 706, the I/O devices 708, and thenetwork devices 710.

The processing unit 702 may be a standard central processor thatperforms arithmetic and logical operations, a more specific purposeprogrammable logic controller (“PLC”), a programmable gate array, orother type of processor known to those skilled in the art and suitablefor controlling the operation of the server computer. Processing unitsare generally known, and therefore are not described in further detailherein. The compute resources 114 can include one or more instances ofthe processing units 702.

The memory 704 communicates with the processing unit 702 via the systembus 712. In some embodiments, the memory 704 is operatively connected toa memory controller (not shown) that enables communication with theprocessing unit 702 via the system bus 712. The memory resources 116 caninclude one or more instances of the memory 704. The illustrated memory704 includes an operating system 714 and one or more program modules716. The operating system 714 can include, but is not limited to,members of the WINDOWS, WINDOWS CE, and/or WINDOWS MOBILE families ofoperating systems from MICROSOFT CORPORATION, the LINUX family ofoperating systems, the SYMBIAN family of operating systems from SYMBIANLIMITED, the BREW family of operating systems from QUALCOMM CORPORATION,the MAC OS, OS X, and/or iOS families of operating systems from APPLECORPORATION, the FREEBSD family of operating systems, the SOLARIS familyof operating systems from ORACLE CORPORATION, other operating systems,and the like.

The program modules 716 may include various software and/or programmodules to perform the various operations described herein. For example,the program modules can include the data-driven feedback control modules122, including the VM fair scheduling control module 202, the real-timeapplication execution state monitoring module 204, the priorityscheduling trigger module 206, the priority scheduling control module208, or some combination thereof. The program modules 716 and/or otherprograms can be embodied in computer-readable media containinginstructions that, when executed by the processing unit 702, performvarious operations such as those described herein. According toembodiments, the program modules 716 may be embodied in hardware,software, firmware, or any combination thereof.

By way of example, and not limitation, computer-readable media mayinclude any available computer storage media or communication media thatcan be accessed by the computer system 700. Communication media includescomputer-readable instructions, data structures, program modules, orother data in a modulated data signal such as a carrier wave or othertransport mechanism and includes any delivery media. The term “modulateddata signal” means a signal that has one or more of its characteristicschanged or set in a manner as to encode information in the signal. Byway of example, and not limitation, communication media includes wiredmedia such as a wired network or direct-wired connection, and wirelessmedia such as acoustic, RF, infrared and other wireless media.Combinations of the any of the above should also be included within thescope of computer-readable media.

Computer storage media includes volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules, or other data. Computer storage media includes, but isnot limited to, RAM, ROM, Erasable Programmable ROM (“EPROM”),Electrically Erasable Programmable ROM (“EEPROM”), flash memory or othersolid state memory technology, CD-ROM, digital versatile disks (“DVD”),or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by the computer system 700. In the claims, the phrase “computerstorage medium” and variations thereof does not include waves or signalsper se and/or communication media.

The user interface devices 706 may include one or more devices withwhich a user accesses the computer system 700. The user interfacedevices 706 may include, but are not limited to, computers, servers,PDAs, cellular phones, or any suitable computing devices. The I/Odevices 708 enable a user to interface with the program modules 716. Inone embodiment, the I/O devices 708 are operatively connected to an I/Ocontroller (not shown) that enables communication with the processingunit 702 via the system bus 712. The I/O devices 708 may include one ormore input devices, such as, but not limited to, a keyboard, a mouse, oran electronic stylus. Further, the I/O devices 708 may include one ormore output devices, such as, but not limited to, a display screen or aprinter.

The network devices 710 enable the computer system 700 to communicatewith other networks or remote systems via a network 718. Examples of thenetwork devices 710 include, but are not limited to, a modem, a radiofrequency (“RF”) or infrared (“IR”) transceiver, a telephonic interface,a bridge, a router, or a network card. The network 718 may include awireless network such as, but not limited to, a Wireless Local AreaNetwork (“WLAN”), a Wireless Wide Area Network (“WWAN”), a WirelessPersonal Area Network (“WPAN”) such as provided via BLUETOOTHtechnology, a Wireless Metropolitan Area Network (“WMAN”) such as aWiMAX network or metropolitan cellular network. Alternatively, thenetwork 718 may be a wired network such as, but not limited to, a WideArea Network (“WAN”), a wired Personal Area Network (“PAN”), or a wiredMetropolitan Area Network (“MAN”). The network 718 may be any othernetwork described herein.

Turning now to FIG. 8, details of a software-defined network (“SDN”) 800are illustrated, according to an illustrative embodiment. Theillustrated SDN 800 is configured in an exemplary SDN configuration andshould not be construed as being limiting in any way.

The illustrated SDN 800 includes a data center node 802 and a pluralityof central office network nodes 804-804N. The illustrated data centernode 802 includes a centralized SDN controller 806. The centralized SDNcontroller 806 can perform on-demand resource allocation tasks on thecentral office network nodes 804-804N on-demand in the control planethrough the NVP 102. The central office network nodes 804-804N canperform hardware reconfiguration tasks in the data plane and virtualizednetwork functions 810 in the control plane. The virtualized networkfunctions 810 can execute at least a portion of the non-real-timeapplications 104 and the real-time applications 106 (best shown anddescribed with reference to FIG. 1).

Based on the foregoing, it should be appreciated that concepts andtechnologies directed to data-driven feedback control system forreal-time application support in virtualized networks have beendisclosed herein. Although the subject matter presented herein has beendescribed in language specific to computer structural features,methodological and transformative acts, specific computing machinery,and computer-readable media, it is to be understood that the conceptsand technologies disclosed herein are not necessarily limited to thespecific features, acts, or media described herein. Rather, the specificfeatures, acts and mediums are disclosed as example forms ofimplementing the concepts and technologies disclosed herein.

The subject matter described above is provided by way of illustrationonly and should not be construed as limiting. Various modifications andchanges may be made to the subject matter described herein withoutfollowing the example embodiments and applications illustrated anddescribed, and without departing from the true spirit and scope of theembodiments of the concepts and technologies disclosed herein.

We claim:
 1. A network virtualization platform comprising: a pluralityof hardware resources; a plurality of virtual machines; and a virtualmachine monitor comprising an application execution state monitoringmodule that, when executed by a compute resource of the plurality ofhardware resources, causes the virtual machine monitor to performoperations comprising waiting for an instruction to track an executionstate of an application, in response to receiving the instruction totrack the execution state of the application, tracking the executionstate of the application, determining whether an application transactionis pending execution, and in response to determining that theapplication transaction is pending execution, recording a state changeof the application from a ready state or a wait state to a run state. 2.The network virtualization platform of claim 1, wherein the operationsfurther comprise determining whether the application transactioncompleted successfully.
 3. The network virtualization platform of claim2, wherein the operations further comprise, in response to determiningthat the application transaction completed successfully: incrementing asuccessful application transaction completion counter; and reporting thesuccessful application transaction completion counter to a priorityscheduling trigger module.
 4. The network virtualization platform ofclaim 3, wherein the virtual machine monitor further comprises thepriority scheduling trigger module that, when executed by a secondcompute resource of the plurality of hardware resources, causes thevirtual machine monitor to perform further operations comprising:receiving the successful application transaction completion counter;determining a critical condition for application transactions;determining whether the critical condition has been met; and in responseto determining that the critical condition has been met, sending anotification to a priority scheduling control module, wherein thenotification notifies the priority scheduling control module to enter apriority scheduling mode.
 5. The network virtualization platform ofclaim 4, wherein the virtual machine monitor further comprises thepriority scheduling control module that, when executed by a thirdcompute resource of the plurality of hardware resources, causes thevirtual machine monitor to perform further operations comprising:receiving, from the priority scheduling trigger module, thenotification; in response to the notification, entering the priorityscheduling mode and resetting a current run state of a virtual machineof the plurality of virtual machines that is executing a non-real-timeapplication to a new ready state before a time-out expires; and settingthe application on the virtual machine to the run state.
 6. The networkvirtualization platform of claim 5, wherein resetting the current runstate of the virtual machine comprises resetting the current run stateof the virtual machine via an interrupt request or signal.
 7. Thenetwork virtualization platform of claim 6, wherein the applicationcomprises a real-time application.
 8. A method comprising: waiting, by avirtual machine monitor comprising an application execution statemonitoring module executed by a compute resource of a plurality ofhardware resources of a network virtualization platform, for aninstruction to track an execution state of an application; in responseto receiving the instruction to track the execution state of theapplication, tracking, by the application execution state monitoringmodule of the virtual machine monitor, the execution state of theapplication; determining, by the application execution state monitoringmodule of the virtual machine monitor, whether an applicationtransaction is pending execution; and in response to determining thatthe application transaction is pending execution, recording, by theapplication execution state monitoring module of the virtual machinemonitor, a state change from a ready state or a wait state to a runstate.
 9. The method of claim 8, further comprising determining, by theapplication execution state monitoring module of the virtual machinemonitor, whether the application transaction completed successfully. 10.The method of claim 9, further comprising, in response to determiningthat the application transaction completed successfully: incrementing,by the application execution state monitoring module of the virtualmachine monitor, a successful application transaction completioncounter; and reporting, by the application execution state monitoringmodule of the virtual machine monitor, the successful applicationtransaction completion counter to a priority scheduling trigger moduleof the virtual machine monitor.
 11. The method of claim 10, furthercomprising: receiving, by the priority scheduling trigger module of thevirtual machine monitor, the successful application transactioncompletion counter; determining a critical condition for applicationtransactions; determining, by the priority scheduling trigger module ofthe virtual machine monitor, whether the critical condition has beenmet; and in response to determining that the critical condition exists,sending, by the priority scheduling trigger module of the virtualmachine monitor, a notification to a priority scheduling control moduleof the virtual machine monitor, wherein the notification notifies thepriority scheduling control module to enter a priority scheduling mode.12. The method of claim 11, further comprising: receiving, by thepriority scheduling control module of the virtual machine monitor, thenotification from the priority scheduling trigger module; in response tothe notification, entering, by the priority scheduling control module ofthe virtual machine monitor, the priority scheduling mode and resettinga current run state of a virtual machine of a plurality of virtualmachines that is executing a non-real-time application to a new readystate before a time-out expires; and setting, by the priority schedulingcontrol module of the virtual machine monitor, the non-real-timeapplication on the virtual machine to the run state.
 13. The method ofclaim 12, wherein resetting the current run state of the virtual machinecomprises resetting the current run state of the virtual machine via aninterrupt request or signal.
 14. The method of claim 13, wherein theapplication comprises a real-time application.
 15. A computer-readablestorage medium having instructions for an application execution statemonitoring module stored thereon that, when executed by a virtualmachine monitor executed by a compute resource of a networkvirtualization platform, cause the virtual machine monitor to performoperations comprising: waiting for an instruction to track an executionstate of an application; in response to receiving the instruction totrack the execution state of the application, tracking the executionstate of the application; determining whether an application transactionis pending execution; and in response to determining that theapplication transaction is pending execution, recording a state changeof the application from a ready state or a wait state to a run state.16. The computer-readable storage medium of claim 15, wherein theoperations further comprise determining whether the applicationtransaction completed successfully.
 17. The computer-readable storagemedium of claim 16, wherein the operations further comprise, in responseto determining that the application transaction completed successfully:incrementing a successful application transaction completion counter;and reporting the successful application transaction completion counterto a priority scheduling trigger module.
 18. The computer-readablestorage medium of claim 17 having further instructions for the priorityscheduling trigger module stored thereon that, when executed by thevirtual machine monitor executed by the compute resource of the networkvirtualization platform, cause the virtual machine monitor to performfurther operations comprising: receiving the successful applicationtransaction completion counter; determining a critical condition forapplication transactions; determining whether the critical condition hasbeen met; and in response to determining that the critical condition hasbeen met, sending a notification to a priority scheduling controlmodule, wherein the notification notifies the priority schedulingcontrol module to enter a priority scheduling mode.
 19. Thecomputer-readable storage medium of claim 18 having further instructionsfor the priority scheduling control module stored thereon that, whenexecuted by the virtual machine monitor executed by the computerresource of the network virtualization platform, cause the virtualmachine monitor to perform further operations comprising: receiving,from the priority scheduling trigger module, the notification; inresponse to the notification, entering the priority scheduling mode andresetting a current run state of a virtual machine of a plurality ofvirtual machines that is executing a non-real-time application to a newready state before a time-out expires; and setting the non-real-timeapplication on the virtual machine to the run state.
 20. Thecomputer-readable storage medium of claim 19, wherein resetting thecurrent run state of the virtual machine comprises resetting the currentrun state of the virtual machine via an interrupt request or signal.